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ABSTRACT 

The  effectiveness  of  small  support  spatial  filters  based  on  mean,  median  and 
morphological  opening  for  the  detection  of  scintillating  ultra-dim  stationary  point 
targets  in  IR  image  sequences  has  been  investigated.  The  effectiveness  of  two  temporal 
filters  was  also  studied.  The  filters  were  applied  to  two  IR  image  sequences;  an  aircraft 
approaching  in  an  uncluttered  background,  and  an  aircraft  receding  in  a  bright  cloudy 
background.  The  spatial  filters  were  effective  in  detecting  the  target  in  the  benign 
background,  but  neither  the  spatial  nor  one  of  the  temporal  filters  were  effective  in  the 
cluttered  environment.  A  combination  of  absolute  frame  differencing  and  small 
support  spatial  filtering  to  correct  for  sensor  motion  was  found  to  give  sufficient 
increase  in  signal  to  clutter  ratio  to  allow  detection. 
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The  Performance  of  Small  Support  Spatial  and 
Temporal  Filters  for  Dim  Point  Target  Detection  in 

IR  Image  Sequences 


Executive  Summary 

Infrared  search  and  track  (IRST)  systems  are  being  developed  to  detect  anti-ship  sea- 
skimming  missiles  at  maximum  possible  range.  The  problem  is  essentially  the 
detection  of  ultra-dim,  stationary,  point-like  targets  in  clutter  in  sequences  of  infrared 
(IR)  images.  This  paper  considers  the  worst  case  when  the  background  consists  of  low 
sunlit  clouds  where  the  brightness  range  is  very  large  and  cloud  edges  would  provide 
many  artefacts  which  could  be  considered  as  possible  targets. 

The  detection  of  ultra-dim  targets  poses  greater  difficulties  than  the  detection  of 
brighter  targets.  A  common  method  of  target  detection  based  on  small  size  spatial 
filters  works  by  predicting  the  background  by  some  type  of  smoothing  operation,  then 
subtracting  the  predicted  background  from  the  original  image.  The  aim  is  to  fit  the 
predicted  background  as  closely  as  possible  in  the  original  image  without  diminishing 
the  target  signal.  Linear  adaptive  matched  filters  can  also  enhance  the  response  to  the 
target.  There  is  a  problem  with  all  spatial  filters  for  detection  of  ultra-dim  point  targets 
with  single  pixel  features  in  strong  clutter  because  there  is  no  spatial  information  in  a 
single  pixel  to  allow  discrimination  from  clutter. 

If  the  target  is  moving,  filters  based  on  the  temporal  profile  of  the  target  as  it  moves 
through  a  pixel  can  be  used.  The  target  for  IRST  has  been  assumed  to  be  stationary,  so 
these  temporal  filters  cannot  be  used.  However,  the  average  brightness  of  the  target  is 
expected  to  increase  over  time  as  it  approaches  the  sensor.  In  the  short  time  frame  there 
will  be  considerable  variation  in  brightness  due  to  scintillation.  These  temporal 
behaviours  may  be  able  to  be  exploited  for  target  detection. 

The  threat  posed  by  anti-ship  missiles  is  of  very  short  duration,  and  requires  response 
in  shortest  possible  time.  Future  staring  IRST  sensor  frame  rates  are  expected  to  be  of 
the  order  of  25-50  frames/s,  and  so  the  images  must  be  processed  in  very  short  times. 
Hence  is  essential  that  detection  algorithms  have  low  computational  load.  The 
operation  of  the  sensor  at  high  frame  rates  allows  the  fact  that  the  target  is  scintillating 
to  be  exploited  to  improve  detectability.  Absolute  frame  differencing  would  suppress 
the  relatively  constant  background  and  highlight  the  rapid  fluctuations  in  the  target 
brightness  cause  by  scintillation.  However,  sensor  jitter  would  also  produce  clutter  in 
the  high  brightness  regions  in  the  image.  A  combination  of  absolute  frame  differencing 
and  small  support  spatial  filtering  to  correct  for  sensor  motion  was  found  to  give 
sufficient  increase  in  signal  to  clutter  ratio  to  allow  timely  detection. 
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1.  Introduction 


Infrared  search  and  track  (IRST)  systems  are  being  developed  to  detect  anti-ship  sea- 
skimming  missiles  at  maximum  possible  range.  Ideally  detection  would  occur  as  soon 
as  the  missile  appeared  over  the  horizon.  The  extreme  range  of  the  target  means  that 
the  missile  target  would  be  sub-pixel  in  size  and  its  angular  motion  would  be  very 
small,  and  it  will  be  assumed  to  be  zero.  The  problem  is  essentially  the  detection  of 
ultra-dim,  stationary,  pointlike  targets  in  clutter  in  sequences  of  infrared  (IR)  images. 
The  case  where  the  background  is  relatively  benign,  such  as  clear  sky  or  very  light 
diffuse  clouds,  has  been  treated  previously [1].  This  paper  considers  the  worst  case 
when  the  background  consists  of  low  sunlit  clouds  where  the  brightness  range  is  very 
large  and  cloud  edges  would  provide  many  artefacts  which  could  be  considered  as 
possible  targets. 

The  detection  of  ultra-dim  targets  poses  considerably  greater  difficulties  than  the 
detection  of  brighter  targets.  "Dim"  will  be  taken  to  mean  low  contrast  with  the 
background,  whether  positive  or  negative.  For  die  case  of  relatively  bright  targets  in 
cluttered  backgrounds,  linear  spatial  filters  are  able  to  decrease  the  magnitude  of  the 
correlated  clutter,  and  increase  the  response  to  a  target.  However,  these  filters  would 
have  a  large  computational  load,  and  may  not  be  practical  for  operation  at  the  frame 
rate  of  an  IRST  system.  Simpler  filters,  including  nonlinear  filters,  may  give 
comparable  results  with  a  lower  computational  load.  There  is  a  problem  with  all  spatial 
filters  for  detection  of  ultra-dun  point  targets  in  strong  clutter  with  single  pixel  features 
because  there  is  no  spatial  information  in  a  single  pixel  to  allow  discrimination  from 
clutter.  If  the  target  is  moving,  filters  based  on  the  temporal  profile  of  the  target  as  it 
moves  through  a  pixel  can  be  used.  The  target  for  IRST  has  been  assumed  to  be 
stationary,  so  these  temporal  filters  cannot  be  used.  However,  the  average  brightness  of 
the  target  is  expected  to  increase  over  time  as  it  approaches  the  sensor.  In  the  short 
time  frame  there  will  be  considerable  variation  in  brightness  due  to  scintillation.  These 
temporal  behaviours  may  be  able  to  be  exploited  for  target  detection. 

The  resolution  of  the  sensor  has  several  effects  on  detectability  of  dim  point  targets.  At 
maximum  range  a  sea-skimming  missile  would  be  smaller  than  the  pixel  size  of 
practical  IRST  sensors,  so  the  signal  from  the  pixel  containing  the  target  would  have  a 
component  from  the  target  and  a  component  from  the  background.  Increasing  the 
resolution  would  have  both  positive  and  negative  effects.  The  target  pixel  at  higher 
resolutions  would  have  a  greater  contribution  from  the  signal  from  the  target, 
increasing  its  effective  brightness.  Increased  resolution  would  also  aid  the  spatial 
separation  of  the  target  and  background.  However  if  temporal  filtering  is  used, 
increased  spatial  resolution  increases  the  effect  of  errors  caused  by  frame 
misregistration  due  to  sensor  motion 

The  threat  posed  by  anti-ship  missiles  is  of  very  short  duration,  and  requires  response 
in  shortest  possible  time.  Future  staring  IRST  sensor  frame  rates  are  expected  to  be  of 
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the  order  of  25-50  frames/ s,  and  so  the  images  must  be  processed  in  very  short  times. 
Hence  is  essential  that  detection  algorithms  have  low  computational  load. 

There  have  been  many  approaches  to  the  problem  of  detection  of  dim  point  targets. 
There  are  a  number  of  linear  adaptive  spatial  filters  such  as  that  of  Soni  et  al  [2],  but 
they  are  computationally  intensive  for  non-stationary  clutter.  Three  dimensional 
spatial- temporal  adaptive  filters  have  been  described  by  Melendez  and  Modestino  [3]. 
A  method  involving  adaptive  thresholding  of  point  intensity  contrast  from 
surrounding  8  pixels  and  accumulating  results  over  95  frames  was  proposed  by  New  et 
al  [4],  Stationary  targets  were  not  considered.  Developments  of  this  method  were  given 
by  Ronda  et  al  [5,  6].  Li  and  Shen  [7]  used  a  frame  summing  after  registration  to  detect 
moving  targets.  A  method  for  clutter  suppression  using  simultaneous  adaptive  spatial- 
temporal  filtering  and  jitter  compensation  has  been  described  by  Tartakovsky  and 
Blazek  [8],  but  it  appears  to  be  computationally  intensive. 

There  have  been  a  number  of  cases  of  the  application  of  temporal  filters  to  dim  point 
target  detection.  Tzannes  and  Brooks  [9]  consider  the  temporal  profile  of  targets  and 
background  as  they  pass  through  a  pixel  location  to  detect  moving  targets.  Yang  et.  al. 
[10]  used  a  combination  of  positive  frame  differencing  and  temporal  averaging  of  the 
frame  difference  to  detect  moving  targets  with  velocities  less  than  0.3  pixels/frame. 

Spatial  filtering  based  on  horizontal  and  vertical  morphological  opening  was  used  by 
Zhu  et  al  [11]  for  small  target  detection.  Max/mean  and  max/median  filters  have  been 
used  by  Deshpande  et  al  [12]  for  moving  point  target  detection. 

None  of  the  filters  considered  above  appear  to  be  ideal  for  the  detection  of  ultra-dim 
scintillating  targets  in  heavy  clutter  at  video  frame  rates.  This  paper  will  examine  the 
effect  of  several  spatial  and  temporal  filters  and  show  that  separately  they  are  not  able 
to  provide  sufficient  detection  capability,  but  it  will  be  suggested  that  a  suitable 
combination  of  spatial  and  temporal  filters  could  provide  acceptable  performance. 

2.  Experimental 

Two  sequences  of  IR  images  will  be  examined  in  this  work.  Both  involve  an  aircraft 
flying  over  Gulf  St  Vincent  off  Adelaide  at  an  altitude  of  approximately  30  m.  The 
effective  target  was  the  surface  of  the  exhaust  pipes  of  the  aircraft.  The  image 
sequences  were  obtained  using  an  Amber  Galileo  midwave  infrared  camera.  The 
Galileo  focal  plane  array  is  256x256  pixels,  each  pixel  30  p  on  side.  The  Galileo  was 
fitted  with  a  300  mm  catadioptric  lens  and  operated  at  25  frames/s.  The  300  mm  lens 
gives  a  pixel  size,  or  instantaneous  field  of  view  (IFOV),  of  0.1  mrad.  The  field  of  view 
(FOV)  was  1.467°. 

The  Gabriel  software  (Aspect  Computing,  Adelaide,  Australia)  was  used  to  display 
image  sequences,  and  also  to  measure  pixel  intensities,  and  means  and  standard 
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deviations  of  image  areas.  Image  filtering  and  processing  were  done  with  specially 
written  C  programs. 

The  Sequence  1  is  the  aircraft  flying  towards  the  sensor  with  a  background  of  very 
diffuse  high  level  clouds.  The  resolution  of  the  original  image  sequence  was  0.1  mrad, 
which  was  higher  than  the  resolution  desired  for  the  application  for  which  the  data 
was  originally  collected.  The  resolution  was  reduced  by  resampling  with  the  average  of 
2x2  pixels  of  the  original  image.  Because  the  aircraft  did  not  fly  a  straight  course,  there 
was  a  drift  in  bearing  during  the  flight  and  the  target  image  moved  over  a  small 
number  of  pixels,  which  is  contrary  to  the  assumption  made  for  a  stationary  target  in  a 
real  IRST.  An  apparently  stationary  target  location  was  produced  using  the  following 
procedure.  The  target  was  manually  tracked  during  the  periods  when  it  was  visible. 
The  values  of  the  target  pixel  and  a  small  surrounding  area  were  copied  to  a  fixed 
location  using  knowledge  of  the  true  track.  The  local  mean  around  the  target  location 
and  the  local  mean  at  the  fixed  location  were  calculated,  and  the  copied  values  were 
adjusted  to  allow  for  the  difference  in  means.  The  intensity  at  the  old  target  location 
replaced  with  the  average  local  intensity.  The  final  frame  of  the  sequence  is  shown  in 
Figure  1. 


Figure  1  Final  frame  of  modified  Sequence  1. 

The  Sequence  2  is  an  aircraft  flying  away  from  the  sensor.  It  has  a  small  but  significant 
angular  velocity,  and  the  target  appears  to  go  in  and  out  of  clear  and  cloud  regions. 
The  resolution  of  the  image  sequence  was  0.1  mrad.  A  second  sequence  was  produced 
by  resampling  with  the  average  of  2x2  pixels  of  the  original  image.  Images  from  both 
sequences  will  be  used  in  this  paper.  Frame  number  1788  of  the  full  resolution 
sequence  is  shown  in  figure  2.  The  isolated  black  and  white  points  are  defective  pixels. 
The  target  is  not  visible  in  this  Figure,  so  an  enlarged  portion  containing  the  target  is 
given  in  Figure  3.  The  target  is  the  brightest  occurrence  in  a  sequence  of  about  10 
frames  about  frame  1788,  when  the  target  can  be  distinguished  from  the  clouds. 

3.  Performance  of  Small  Size  Spatial  Filters 

A  common  method  of  target  detection  based  on  small  size  spatial  filters  works  by 
predicting  the  background  by  some  type  of  smoothing  operation,  then  subtracting  the 
predicted  background  from  the  original  image.  The  aim  is  to  fit  the  predicted 
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Figure  3.  Frame  1788,  area  containing  the  target. 


background  as  closely  as  possible  in  the  original  image  without  diminishing  the  target 
signal.  Linear  adaptive  matched  filters  can  also  enhance  the  response  to  the  target.  The 
behaviour  of  several  small  support  spatial  filters  will  be  analysed  to  determine  their 
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positive  and  negative  features  in  order  to  indicate  their  likely  usefulness  in  various 
situations. 

A  horizontal  section  through  the  target  from  the  last  frame  in  the  sequence  of  clear 
background  images  is  given  in  Figure  4.  The  one  pixel  wide  target  is  clearly  visible. 
Horizontal  sections  through  the  target  from  the  full  and  half  resolution  images  of  the 
cloud  background  sequence  are  given  in  Figure  5.  Pixel  points  are  not  shown  in  the 
cloud  sequences  for  clarity.  The  scale  of  the  half  resolution  section  has  been  expanded 
by  a  factor  of  2  in  this  figure.  The  target  is  a  single  pixel  in  both  images,  and  its 
magnitude  is  small.  The  magnitude  of  the  half  resolution  target  is  approximately  a 
quarter  of  the  full  resolution  image,  as  expected.  It  can  be  seen  that  there  is  very  little  to 
distinguish  the  target  from  other  peaks  in  the  Vi  scale  image. 

The  ability  of  several  spatial  filters  to  highlight  possible  targets  will  be  investigated. 
The  classes  of  filters  considered  are  based  on  morphological  elements,  two  types  of 
means,  and  medians.  In  this  discussion  only  positive  contrast  targets  will  be 
considered,  but  an  analogous  procedure  could  be  used  for  negative  contrast  targets. 

The  application  of  the  filters  will  be  illustrated  by  application  to  ID  sections  of  the  half 
resolution  of  the  clear  background  IR  image  shown  in  Figure  4  and  the  full  resolution 
image  section  shown  in  Figure  5. 


Figure  4.  Rozv  8  of  frame  1625  of  clear  background  image  sequence 
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—  Original 
— 1/2  Resolution 


Figure  5.  Row  157  of frame  1788  of  cloud,  background  image  sequence 


3.1  Median  filters 


The  results  of  applying  a  3  element  linear  median  filter  to  the  ID  sections  of  the  ima 
in  Figures  4  and  5  are  given  in  Figures  6  and  7.  As  expected,  the  median  curve  is 


Original 
—  Median 


Figure  6.  Original  image  and  median  of  clear  background  scene 
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intermediate  between  local  extrema,  and  in  the  strong  clutter  the  median  overlays  the 
image  in  the  large  slope  regions.  The  curve  of  the  difference  between  the  original 
image  and  the  median  would  consist  of  positive,  negative  and  zero  points,  with  a  mean 
near  zero. 

3.2  Mean  filters 

The  mean  filter  was  defined  as  the  mean  of  the  values  of  the  two  adjacent  pixels.  The 
results  of  applying  this  filter  to  the  clear  and  cloud  background  1  D  image  sections  are 
given  in  Figures  8  and  9. 

The  results  are  generally  similar  to  those  for  the  median  filter,  but  there  is  a  difference 
between  the  curves  in  the  regions  of  high  gradients.  The  mean  shows  both  positive  and 
negative  deviations  from  the  original  curve  in  the  high  slope  clutter  regions,  but  in 
addition  the  points  adjacent  to  peaks  are  also  affected. 


—  Original 
- Mean 


Figure  9.  Original  image  and  mean  of  cloud  background  scene. 

3.3  Two  element  open  filter 

The  previous  filters  effectively  spanned  a  linear  range  of  3  pixels.  A  filter  spanning 
only  two  pixels  would  be  expected  to  follow  the  original  curve  more  closely.  A  two 
pixel  range  filter  can  be  defined  as  the  opening  filter,  Orl2.  The  standard  mathematical 
morphology  open  operation  is  defined  as  an  erosion  followed  by  a  dilation  with  the 
reflection  about  the  origin  of  the  same  structuring  element  (SE).  The  effect  of  applying 
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the  Orl2  filter  to  the  ID  clear  and  cloud  image  sections  is  illustrated  in  Figures  10  and 

11. 


Original 
—  Orl2 


Figure  10.  Original  image  and  the  Orl2  filter  output  of  clear  background  scene 


Figure  11.  Original  image  and  the  Orl2  filter  output  of  cloud  background  scene 


—  Original 

—  Orl2 
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The  Orl2  filter  highlights  all  tire  local  maxima,  and  the  output  is  never  locally  greater 
than  the  original  image.  The  filter  follows  the  original  in  the  high  slope  regions 
similarly  to  the  median  filter. 

10  n 


Column  number 

Figure  12.  Difference  between  the  original  image  and  median  and  Orl2  filter  outputs  for  the 
clear  background 


0  50  100  150  200  250 


Column  numnber 

Figure  13.  Difference  between  the  original  image  and  median  and  Orl2  filter  outputs  for  the 
cloud  background 
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3.4  Comparison  of  spatial  filters 

Filters  for  highlighting  local  peaks  can  be  obtained  by  subtracting  the  filter  output  from 
the  original  image.  The  effects  of  subtracting  the  ID  median  and  Orl2  filters  for  the 
clear  and  cloud  backgrounds  are  illustrated  in  Figures  12  and  13. 

The  target  peak  in  the  clear  background  section  is  well  highlighted  by  both  filters,  but 
the  target  peak  in  the  cloud  background  section  is  only  slightly  more  prominent  than 
other  peaks  not  due  to  the  target. 

The  non-negative  output  from  both  filters  is  identical  for  both  backgrounds,  but  the 
median  filter  also  gave  a  negative  output.  Note  that  the  scale  for  the  clear  background 
is  more  expanded  than  the  cloud  background.  The  computational  load  of  the  Orl2  filter 
is  less  than  that  of  the  median  filter,  so  where  only  the  positive  output  is  required,  the 
Or  12  filter  is  to  be  preferred.  The  filter  process  can  be  inverted  for  negative  contrast 
only  situations. 

There  are  regions  of  zero  output  for  both  filters,  but  they  are  longer  in  the  cloud  clutter 
image,  and  they  usually  correspond  to  high  slope  regions  in  the  original  image.  This 
observation  has  important  implications  for  the  use  of  median  filters  in  the  detection  of 
ultra-dim  targets.  If  a  target  occurs  in  a  high  slope  region,  a  median  filter  will  not 
highlight  it  unless  the  pixel  value  is  the  maximum  in  the  range  of  the  median  filter. 
This  means  that  ultra-dim  targets  could  be  missed  in  strong,  spatially  rapidly  varying 
clutter.  The  mean  filter  does  not  suffer  from  this  drawback. 

The  effects  of  subtraction  by  the  mean  and  Orl2  filters  are  shown  in  Figures  14  and  15. 
The  target  peak  in  the  clear  background  section  is  well  highlighted  by  both  filters,  but 
the  target  peak  in  the  cloud  background  section  is  lost  in  a  large  number  of  peaks  in  the 
mean  filter  output.  The  positive  parts  are  very  similar  for  the  clear  background  case 
because  the  background  has  no  high  slope  regions.  However,  in  the  cluttered  case  the 
mean  filter  highlights  many  more  positive  peaks.  This  filter  would  be  more  effective  in 
highlighting  targets  in  high  slope  regions.  However,  where  there  is  significant 
curvature  in  the  background  the  mean  filter  would  give  peaks  proportional  to  the 
amount  of  curvature.  If  the  background  did  not  vary  in  time  these  peaks  would  appear 
as  false  targets.  The  mean  filter  would  be  more  effective  at  highlighting  targets,  but  at 
the  expense  of  a  very  much  larger  number  of  false  alarms.  This  number  of  false  alarms 
may  overwhelm  any  practical  tracking  system. 

In  the  benign  clear  background  case  the  small  support  spatial  filters  gave  good 
discrimination  between  the  target  pixels  and  the  background,  even  though  the  targets 
were  ultra-dim.  However,  in  strong  clutter  backgrounds  none  of  the  filters  alone  is 
capable  of  reliably  detecting  ultra-dim  targets.  In  addition,  since  a  real  target  may 
appear  anywhere  in  the  image,  it  might  not  necessarily  produce  a  peak,  and  hence 
would  not  be  highlighted  by  a  spatial  filter.  For  example,  a  target  may  appear  in  the 
bottom  of  a  trough  in  the  brightness  curve,  which  would  only  have  the  effect  of 
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reducing  the  depth  of  the  trough  and  not  appearing  as  a  peak.  Even  if  the  target 
produced  a  peak,  there  is  no  way  of  determining  if  the  peak  is  due  to  a  target  or  clutter. 
Another  serious  problem  is  that  for  all  filters  in  the  cloud  backgrounds  there  are  many 
false  detections,  particularly  for  the  mean  filter. 

These  considerations  show  that  for  the  case  of  targets  in  benign  backgrounds  the 
morphological  and  mean  filters  may  give  acceptable  performance.  However,  spatial 
filters  alone  cannot  be  used  for  reliable  detection  of  ultra-dim  targets.  However,  they 
may  be  useful  in  combination  with  other  types  of  filters.  This  aspect  will  be  examined 
in  Section  5. 

3.5  2D  implementation 

The  discussion  so  far  has  only  considered  ID  filters,  but  most  image  processing  is  2 
dimensional.  Median  filtering  in  ID  is  well  understood,  but  the  generalisation  to  2D  is 
not  straight  forward.  Arce  and  McLoughlin  [13]  showed  that  the  median  filter  based  on 
the  standard  square  window,  which  is  often  used,  gave  a  greater  loss  of  resolution  of 
small  features  in  an  image  than  the  max/median  filter.  The  max/median  filter  is 
defined  as  the  maximum  of  a  series  of  linear  median  filters  through  the  point  of 
interest.  This  means  that  the  max/ median  filter  follows  the  original  image  more  closely 
than  the  square  window  median,  and  when  subtracted  from  the  original  image  the 
result  would  reduce  the  positive  noise  which  would  have  to  be  considered  as  possible 
point  targets. 

A  mean  filter  for  the  2D  case  could  be  defined  as  the  mean  value  of  the  surrounding  8 
pixels  for  each  pixel.  However,  a  filter  which  would  give  a  closer  fit  to  the  original 
image  would  be  the  max/mean  filter.  The  max/mean  filter  is  defined  in  an  analogous 
way  to  the  max/median  filter,  as  the  maximum  of  a  series  of  linear  mean  filters 
through  the  point  of  interest.  In  thelD  case  the  average  of  the  output  of  the  mean  filter 
for  the  clear  background  case  will  be  zero.  In  the  2D  case  where  the  max/ mean  filter  is 
used,  the  output  would  be  biased  in  the  positive  direction,  and  the  average  of  the 
output  would  be  greater  than  zero. 

Max/mean  and  max/median  filters  have  been  used  by  Deshpande  et  al  [12]  for 
moving  point  target  detection. 

A  max/Or!2  filter  for  2D  can  be  defined  analogously  to  the  max/ median  filter. 

Examples  of  the  application  of  max/ median,  max/ opening  and  max/ mean  filters  will 
be  given  in  Section  5,  where  combined  spatial  and  temporal  filtering  will  be  discussed. 
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4.  Performance  of  Temporal  Filters 


4.1  Deviation  from  temporal  mean 

One  possible  way  of  detecting  a  target  is  to  record  when  the  temporal  behaviour  of  a 
pixel  begins  to  deviate  from  that  of  its  neighbours.  There  are  a  number  of  factors 
affecting  pixel  brightness  that  are  not  due  to  targets  which  must  be  allowed  for  if  this 
approach  is  used.  Change  in  ambient  lighting,  ie  the  sun  coming  out,  or  drift  in  sensor 
electronics  are  possibilities. 

The  Sequence  1  dataset  will  be  used  to  illustrate  the  effect  of  some  temporal  filters.  The 
background  above  the  horizon  consists  of  very  diffuse  high  level  clouds,  which  are 
essentially  featureless.  Figure  1.  There  is  no  target  motion  so  each  pixel  operates  as  a 
separate  sensor.  The  target  appears  at  pixel  40,8,  and  pixels  at  45,8  and  60,12  will  be 
used  to  indicate  the  background.  The  brightness  values  of  the  pixels  in  the  sequence 
are  given  in  Figure  16. 


Figure  16.  Pixel  values  of  target,  (40,8);  and  background  pixels 


It  can  be  seen  that  after  about  frame  800  all  signals  start  to  increase,  but  the  target 
increases  more  rapidly.  The  reason  for  the  increase  in  the  background  is  not  known,  it 
may  be  the  sun  coming  from  behind  a  cloud.  The  greater  scatter  of  the  target  intensity 
is  due  to  scintillation. 
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One  possible  way  to  detect  the  appearance  of  the  target  would  be  to  calculate  the 
deviations  from  a  temporal  running  average  and  declare  a  target  when  the  signals  are 
above  a  threshold  value  from  the  average.  For  ease  of  computation  a  running  average 
defined  below  will  be  used. 

where  M(k)  is  the  pixel  mean  at  frame  k,  F  is  the  effective  number  of  frames  included 
in  the  mean,  and  C  is  the  current  pixel  value. 

Plots  of  deviation  of  the  current  target  value  from  the  current  mean  are  given  in 
Figures  17a-17c.  It  can  be  seen  that  the  trend  of  the  deviation  from  the  mean  is  zero, 
independent  of  the  number  of  effective  values  in  tire  mean.  The  same  plots  for  the 
background  are  given  in  Figures  18a-18c. 


Figure  17a.  Deviation  of  target  pixel  value  from  running  mean  averaged  over  4  frames. 
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Figure  17b.  Deviation  of  target  pixel  value  from  running  mean  averaged  over  32  frames. 
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Figure  17c.  Deviation  of  target  pixel  value  from  running  mean  averaged  over  128  frames. 
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Figure  18a.  Deviation  of  background  pixel  value  from  running  mean  averaged  over  4  frames. 
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Figure  18b.  Deviation  of  background  pixel  value  from  running  mean  averaged  over  32  frames. 


Figure  18c.  Deviation  of  background  pixel  value  from  running  mean  averaged  over  128  frames. 


The  deviations  from  the  mean  for  the  background  pixel  show  constant  scatter  over  the 
whole  range  of  frames,  but  the  deviations  for  the  target  pixel  show  increasing  scatter 
when  the  target  is  present.  In  both  cases  the  average  deviation  is  approximately  zero. 

Hence  long  term  deviations  in  pixel  intensity  from  a  running  mean  over  time  cannot  be 
used  to  detect  ultra-dim  targets,  but  it  may  be  possible  to  use  the  short  term  scatter. 
This  possibility  will  not  be  considered  here. 

4.2  Target  scintillation 

In  most  cases  the  target  will  be  scintillating  and  the  background  will  be  approximately 
constant  in  time.  Hence  the  difference  in  temporal  behaviour  of  the  target  and 
background  could  be  used  for  target  detection.  The  temporal  variation  of  the  image  can 
be  quantified  by  taking  the  absolute  values  of  differences  of  corresponding  pixel  values 
in  consecutive  frames.  A  plot  of  absolute  frame  differences  for  the  target  and 
background  pixels  from  the  Sequence  1  dataset  is  given  in  Figure  19.  The  values  were 
averaged  with  a  running  mean  of  32  frames  to  increase  clarity  of  the  figure. 
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Figure  19.  Absolute  frame  differences  averaged  over  32  frames,  target  pixel  -  (40,8)  and 
background  (45,8) 

It  can  be  seen  that  the  difference  is  about  1  count  at  frame  800,  and  up  to  3  counts  at  the 
end  of  the  sequence.  This  difference  should  be  sufficient  to  enable  detection. 

Before  testing  this  method  on  the  full  frame  Sequence  1  data,  a  complication  to  the 
method  in  cluttered  scenes  will  be  discussed  in  the  next  section. 

5.  Combined  Temporal  and  Spatial  Filtering 

If  the  frames  in  an  image  sequence  are  highly  accurately  registered  then  taking  the 
absolute  value  of  frame  differences  alone  would  provide  a  possible  method  of 
detection  of  scintillating  targets.  However,  the  sensor  is  likely  to  be  subjected  to  a 
variety  of  motions  including  random  vibrations  and  platform  motion.  The  stabilisation 
is  unlikely  to  be  perfect  and  some  image  movement  would  occur  between  frames. 
When  taking  frame  differences  this  would  cause  a  non-zero  response  to  occur  in 
regions  of  high  brightness  slope.  These  responses  could  appear  as  false  targets. 
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Figure  20.  Frame  42  of  absolute  frame  differencing  of  Sequence  2. 

The  effect  of  image  motion  on  the  absolute  frame  difference  (AFD)  of  an  image 
sequence  is  illustrated  in  Figure  20.  This  figure  is  frame  42  of  the  AFD  of  the  Sequence 
2  data,  and  is  a  result  of  a  wind  gust  disturbing  the  camera.  It  is  an  example  of  large 
subpixel  misregistration.  Subpixel  motions  only  need  be  considered  as  other  methods 
can  be  used  to  register  to  a  resolution  of  1  pixel.  The  resolution  of  this  image  is  0.1 
mrad.  It  can  be  seen  that  large  areas  of  the  image  have  significant  values,  even  near  the 
horizon.  The  pixel  value  due  to  the  target  is  99,  and  maximum  value  in  the  frame  is  256 
due  to  a  sun  glint  on  the  ocean.  The  maximum  value  in  the  clouds  is  65.  This  clutter 
must  be  suppressed  to  allow  the  target  to  be  detected. 

The  spatial  variation  of  the  frame  differences  of  the  background  is  correlated,  since  the 
image  moves  as  a  whole.  Advantage  can  be  taken  of  this  fact,  and  also  the  fact  that  the 
temporal  variations  of  target  scintillations  and  background  are  temporally 
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uncorrelated.  The  spatial  correlation  only  extends  to  adjacent  pixels  so  an  opening 
operation  with  a  2  element  structuring  element  aligned  in  the  direction  of  motion 
should  closely  follow  the  intensity  distribution  of  the  blurring  caused  by  the  motion. 
Subtraction  of  the  AFD  operated  on  by  a  max/opening  filter  from  the  original  AFD 
should  remove  most  of  the  effect  of  motion,  and  leave  the  single  point,  scintillating, 
target.  The  Orl2  filter  applied  the  frame  42  of  the  AFD  sequence  is  given  in  Figure  21, 
which  shows  a  considerable  decrease  in  clutter  from  Figure  20.  The  effect  of  a 
max/mean  subtraction  filter  on  the  AFD  is  given  in  Figure  22,  which  shows  the  non- 
negative  values  in  the  frame.  It  can  be  seen  that  there  is  a  considerable  reduction  in  the 
magnitude  of  the  clutter  in  this  case  as  well. 


Figure  22.  Or/2  filter  output  subtracted  from  the  absolute  frame  difference  of  frame  42  of 
Sequence  2. 

To  compare  the  action  of  the  different  filters  on  the  clutter,  plots  of  the  filter  output 
along  row  96  of  the  frames  are  given  in  Figures  23-24.  This  row  includes  many  cloud 
edges,  and  illustrates  the  possible  severity  of  the  effect  of  sensor  motion. 
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Figure  22.  Max/mean  filter  output  subtracted  from  the  absolute  frame  difference  of  frame  42  of 
Sequence  2. 


The  curve  for  the  AFD  in  the  figures  shows  only  small  values  for  the  first  60  columns, 
then  large  differences  for  the  rest  of  the  row,  as  would  be  expected  from  the  image 
frame.  Figure  20.  The  target  value  for  the  AFD  frame  is  99,  and  for  the  Orl2  frame  is  9. 
The  max/Orl2  curve  follows  the  AFD  curve  in  the  cloud  free  area,  but  for  the  rest  of 
the  row  it  shows  only  a  few  small  positive  peaks,  with  many  zero  values. 
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Figure  23.  Pixel  values  along  row  96  of  frame  42  of  Sequence  2.  Absolute  frame  difference 
(AFD)  and  Orl2  subtraction  filter. 
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Figure  24.  Pixel  values  along  row  96  of  frame  42  of  Sequence  2.  Absolute  frame  difference 
(AFD)  and  mean  subtraction  filter. 
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Figure  25.  Pixel  values  along  row  96  of  frame  42  of  Sequence  2.  Absolute  frame  difference 
(AFD)  and  max/mean  subtraction  filter. 


The  curve  for  the  mean-8  subtraction  filter,  where  the  mean  is  taken  from  the  8 
surrounding  values,  is  given  in  Figure  24.  It  follows  the  AFD  curve  for  the  first  60 
columns,  and  then  shows  a  similar  response,  but  with  values  about  half  the  AFD 
values.  This  would  not  appear  to  give  sufficient  reduction  in  clutter,  and  this  is 
confirmed  by  observation  of  the  filtered  image  sequence. 

The  max/mean  filter  gives  a  significant  reduction  in  response  to  the  clutter.  Figure  25. 
There  is  a  much  reduced  number  of  positive  values  and  these  are  much  lower  than  the 
AFD  frame.  The  target  value  for  the  filtered  output  in  this  frame  is  52,  which  is  half  the 
AFD  value.  Hence  the  max /mean  filter  appears  to  be  suitable  for  increasing  the 
detectability  of  scintillating  targets. 

The  relative  efficiency  of  the  max/mean  and  Orl2  filters  in  reducing  the  motion 
induced  clutter  of  the  AFD  sequence  was  measured  over  2  short  sequences.  The  first 
sequence  was  frames  41-51,  where  the  target  was  relatively  bright,  and  the  second 
frames  1360-1380,  where  the  target  was  more  difficult  to  detect.  For  each  sequence  the 
average  value  of  the  target  pixel  was  determined,  and  the  clutter  was  measured  both 
by  the  average  number  of  non-zero  pixels,  and  by  the  average  value  of  the  clutter 
pixels.  The  results  are  given  in  Table  1.  Both  filters  reduced  the  average  target  value  by 
approximately  half,  but  the  clutter  measures  were  reduced  by  a  much  greater  factor, 
which  considerable  increased  the  signal  to  clutter  ratio.  A  measure  of  the  signal-to- 
clutter  ratio  (SCR)  can  be  defined  as  the  ratio  of  the  target  signal  to  average  clutter  pixel 
value.  Table  2  gives  SCR  values  for  the  AFD  and  the  filter  outputs,  and  the  SCR  gain 


23 


DSTO-TR-1282 


for  the  filters.  It  can  be  seen  that  both  filters  produce  a  considerable  increase  in  SCR, 
with  a  greater  increase  of  a  factor  of  about  6  being  produced  by  the  Orl2  filter. 


Table  1.  Performance  of  max/mean  and  Orl2  subtraction  filters  for  clutter  reduction  of  absolute 
frame  difference  (AFD)  images. 


Target 

value 

Number  of 
active 
pixels 

Average 

pixel 

value 

Frames  41-51 

u  AFD 

87.50 

56565 

3.19 

Orl2 

33.36 

4940 

0.18 

Max/ mean 

39.90 

12015 

0.48 

Ratio  Orl2:  Max/ mean 

0.84 

0.41 

0.39 

Frames  1360-1380 

AFD 

30.29 

55234 

2.43 

Orl2 

12.95 

5056 

0.18 

Max/ mean 

19.29 

11736 

0.42 

Ratio  Or  12: Max/ mean 

0.67 

0.43 

0.42 

Table  2.  Signal-to-clutter  ratio  and  signal-to-clutter  ratio  gain  from  max/mean  and  Orl2  filters 


SCR 

SCR  gain 

Frames  41-51 

Abs 

27.4 

Orl2 

185.3 

6.8 

Max/ mean 

83.1 

3.0 

Ratio  Orl2:Max/mean 

2.2 

Frames  1360-1380 

Abs 

12.5 

Orl2 

71.9 

5.8 

Max  /mean 

45.9 

3.7 

Ratio  Or  12:  Max/ mean 

1.6 

A  more  detailed  study  with  synthetic  data  is  required  to  fully  quantify  the  effect  of  the 
filters  and  determine  their  suitability  for  ultra-dim  target  detection. 
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6.  A  Test  of  Detection  by  Frame  Differencing  and 
Max/Mean  Filtering 

A  test  of  the  ability  of  the  combined  spatial  and  temporal  filters  to  detect  an  ultra-dim 
was  carried  out  using  the  Sequence  1  data.  Since  the  target  in  Sequence  2  was  not 
stationary  or  of  constant  strength,  it  was  not  possible  to  determine  the  target  statistics 
in  that  sequence.  However,  Sequence  1  provided  a  known  target  and  constant 
background. 

Because  the  target  is  scintillating,  a  detector  based  on  a  simple  threshold  is  likely  to 
perform  poorly.  To  take  account  of  the  scintillation,  the  Bayesian  track  before  detect 
algorithm  from  [1]  was  used  as  the  target  detector.  In  order  to  calculate  a  likelihood 
ratio,  the  distribution  function  of  the  background  values  has  to  be  known,  and  the 
process  is  much  simpler  if  that  function  is  a  Gaussian.  An  estimate  of  the  background 
statistics  was  obtained  from  the  values  of  pixel  (128,4)  over  the  whole  sequence,  which 
appears  to  be  clear  sky.  The  histograms  of  the  AFD  and  filtered  values  for  the  whole 
sequence  are  given  in  Figure  26.  The  AFD  values  are  all  non-negative,  by  definition, 
and  the  histogram  peaks  at  1  count  The  Orl2  curve  is  very  asymmetrical  and  peaks  at 
zero  as  expected  from  Figure  23,  (the  zero  value  is  suppressed  because  it  is  far  off 
scale).  The  mean  curves  are  more  symmetrical  and  bell  shaped.  A  fair  fit  to  a  Gaussian 
curve  was  obtained  for  the  max/mean  curve.  The  mean  was  -1.41  and  the  standard 
deviation  was  1.89,  with  a  correlation  coefficient  of  0.991 
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Figure  26.  Histograms  of  values  of  pixel  128,4  of  Sequence  2  for  the  absolute  frame  difference , 
and  filters  Orl2 ,  mean  and  max/mean. 
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The  output  of  the  absolute  frame  differencing  followed  by  the  max/mean  filter  for  the 
target  pixel  is  given  in  Figure  27.  There  are  areas  of  increased  value  after  frame  750 
when  the  target  is  present.  The  filtering  produced  very  little  change  in  the  values  from 
the  AFD  values.  There  is  a  small  but  significant  increase  in  filter  output  at  the  target 
position  when  the  target  appears,  but  the  difference  is  so  small  that  the  extra  sensitivity 
of  the  Bayesian  algorithm  is  required  to  reliably  detect  it.  The  algorithm  was  set  up  for 
the  max/mean  filter  because  the  background  statistics  were  Gaussian,  which  gave  a 
simple  log  likelihood  ratio. 

The  result  of  the  detection  algorithm  is  illustrated  in  Figure  28.  The  curve  plots  the 
output  value  for  the  target  pixel.  Any  value  above  zero  corresponds  to  a  detection.  It 
can  be  seen  that  the  target  is  detected  for  most  of  the  time  it  is  present,  but  the  number 
of  detections  is  less  than  for  the  direct  detection  from  the  original  image  [1],  This  is  to 
be  expected  as  some  information  is  lost  by  the  extra  clutter  minimisation  procedures. 
There  were  some  false  alarms,  caused  mainly  by  birds.  Optimisation  of  the  parameters 
in  the  algorithm  would  probably  reduce  the  false  alarm  rate. 

Because  of  the  loss  of  information  in  the  calculating  the  AFD  and  applying  the 
temporal/ spatial  filtering,  this  procedure  would  not  be  the  most  effective  detection 
algorithm  for  bright  targets  or  benign  backgrounds.  In  practice  it  would  be  necessary 
to  segment  an  image  and  apply  the  AFD  and  temporal/ spatial  filter  to  cluttered 
regions,  and  more  standard  spatial  and/or  temporal  filters  to  uncluttered  regions. 
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Figure  27.  Output  of  frame  differencing  plus  max/mean  filter  for  the  target  pixel  40,8  for  the 
Sequence  l  data. 
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Figure  28.  Output  from  detection  algorithm  for  target  pixel  40,8. 

Further  work  is  required  to  determine  the  range  of  conditions  over  which  the 
temporal/ spatial  filter  gives  better  results  than  standard  filters. 


7.  Conclusions 

The  effectiveness  of  small  support  spatial  filters  based  on  mean,  median  and 
morphological  opening  for  the  detection  of  scintillating  ultra-dim  stationary  point 
targets  in  IR  image  sequences  has  been  investigated.  The  effectiveness  of  two  temporal 
filters  was  also  studied.  It  was  demonstrated  that  spatial  or  temporal  filters  used 
separately  are  not  effective  for  the  detection  of  these  targets  in  heavy  clutter  in  IR 
image  sequences,  but  the  spatial  filters  perform  adequately  for  benign  clutter. 

For  image  sequences  at  video  frame  rates  a  detection  algorithm  based  on  taking 
absolute  frame  differences  followed  by  small  support  spatial  filter  can  give  a  significant 
increase  in  signal  to  clutter  ratio  of  scintillating  targets  in  heavy  clutter. 
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